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SYSTEM AND METHOD FOR SEAMLESS SWITCHING 

CROSS-REFERENCE TO RELATED APPLICATIONS 

This patent application claims priority from U.S. Provisional Application Serial No. 
60/236,990 filed September 29, 2000 by Haberman et al, which is incorporated herein by 
reference. 

FIELD OF THE INVENTION 

This disclosure relates, generally, to a system and method for transmitting data and, more 
particularly, to a system and method of seamless switching between a plurality of data streams. 

BACKGROUND 

Typical television broadcasts do not allow personalization of television content to an end 
user's profile. The standard television broadcast provides only one variant of every channel. 
The channel is selected by the viewer and the STB (set top box) selects the video and audio for 
that channel from the broadcast material. While this system allows the viewer to select their 
favorite channel or show from the available set, the individual viewer will be watching exactly 
the same content as everyone else that selects that channel. 

Due to the fact that channels are created to attract a wide range of viewers, viewers 
typically have different preferred channels. Disadvantageous^, this is particularly evident when 
a program being broadcast on a channel is interrupted by a commercial advertisement that does 
not appeal to the interests of the viewer. The inevitable result is that the viewer will switch to 
another channel to avoid watching that particular commercial advertisement. It would be 
advantageous to personalize channels to be viewed by a viewer tailored to their particular 
interests and personal situations, including personalized ads or messages which make such ads 
more relevant to the viewer. 

A method of creating personalized messages is disclosed in co-pending U.S. Patent 
Application 09/545,015 filed on April 7, 2000, which is incorporated herein by reference. 



Personalized versions of advertisements or entire programs can be created on a group or 
individual level. One technique which can assist in the process of assembling personalized 
messages is the ability to switch rapidly between multiple data streams (such as audio and/or 
video feeds), in order to assemble the message in real time, possibly as the message is being 
5 viewed by the end user. 

However, switching between multiple data streams is problematic. One problem is that 
switching may not be instantaneous. With present technology, it is difficult to switch from one 
high-bandwidth data stream to another without missing some data during the time the switch is 
switching. This is true no matter what the type of data stream is, including analog signals, 

10 digitized signals, digital data, multiplexed digital data etc., or the type of switch, whether 

electromechanical, solid state, analog or digital demultiplexer, or a software switch. Also, timing 
the switch point to minimize data interruption is very difficult. The act of switching typically 
results in a switch gap, a period of no data during the interim while the switch is switching 
between data streams. For multimedia (such as television) signals, switching introduces audible 

1 5 and visible artifacts in the sound and picture. 

An example of this switching problem also occurs with MPEG-2 encoding. MPEG-2 
defines a broadcast quality standard for multimedia signals, based on digital technology. The 
MPEG-2 standard includes the capability for compressing, coding and transmitting high-quality 
multi-channel, multimedia signals over broadband networks. MPEG-2 defines standards for 
20 program streams and transport streams. The program stream provides video, audio and data 

elements that have a common time base. The transport stream combines several programs into a 
single stream, wherein the programs may not have a common time base. MPEG-2 encodes 
media signals as frames, and switching between MPEG-2 encoded transport streams requires 
frame-accurate switching to avoid switching artifacts. 

25 Fig. 1 shows switching and decoding components of an example digital television 

receiver 20. The transport stream 48 carrying multiple encoded data streams enters the demux 
(demultiplexer) 32. This demux 32 serves as the switch, by selecting which data stream in the 
multiplexed transport stream to pass on. The selected data stream includes both a video stream 
and an audio stream. These streams are then decoded respectively by a video decoder 42 and an 
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audio decoder 44. The video decoder 42 and audio decoder provide buffering on these input 
streams. The results of the decoding are a stream of video frames 40 and audio samples 38, 
which can then be sent to display and audio equipment. The decoder is controlled by a receiver 
controller 46, which typically uses a microprocessor and software. 

5 Normally, when switching between different video 40 and audio tracks 38 within the 

Transport Stream 48, the receiver controller 46 first mutes/blanks the affected decoder, (as 
shown by arrows 43 and 45, then switches the Demux 32 settings and then unmutes/unblanks the 
decoder(s). This will present a moment of silence/black to the viewer. It will never be a seamless 
switch for the viewer. 

10 In an attempt to get a seamless switch, the mute/unmute sequence may be skipped. Now, 

however, the results depend on the exact moment of the switch with respect to the incoming data 
from the transport stream. Digital compression and transmission creates interdependences 
between groups of video frames because of encoding and packaging and groups of audio samples 
because of packaging. Only at certain points within the data stream of the transport stream is it 

15 safe (W.R.T. visible and audible artifacts) to switch out of the current stream or into another 
stream. The requirement of exactly hitting a combination of safe exit and safe entry point make 
the seamless switch very difficult. In addition to this, the decoders 42, 44 are typically the only 
devices in the receiver 20 that can detect the right moment, while the demux 32 is the device that 
must be switched. Because of extensive data buffering between the demux and the decoders, 

20 detection by the decoder is of no use to determine the right moment to switch the demux. 

It may be possible to add a detection device to the demux that would duplicate part of the 
task of the decoder with a defined timing relation to the transport stream data. This would allow 
detection of safe exit points. However, this solution requires extra hardware, and thus is 
incompatible with all existing receivers. 

25 Because of the way that digital television receivers are constructed, the switching 

mechanism that allows the selection of video and audio streams from the transport stream is 
located before the playback buffering. And because the data is transmitted in burst-mode, there is 
no way to know the fill level of the buffer without detailed knowledge of the incoming transport 
stream, and the current playback time of the receiver. 
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One solution would be to build new receivers with specialized hardware and software 
(including buffering at several locations in the receiver) to support such switching. However, 
this solution increases the cost and complexity of receivers, and can not take advantage of the 
existing receivers on the market. 

5 SUMMARY 

In accordance with the present invention, there is provided a transmission system and 
method for seamlessly switching between a plurality of data streams to produce an output data 
stream with minimal or without any switching artifacts. Preferably, the seamless switch includes 
no visual or audible artifacts of the switch during reproduction of data. The disclosed 
10 transmission system and method for seamless switching may be utilized in applications including 
slot based broadcasts where frame and sample accurate switching in a digital television 
environment is required. The system and method can facilitate multi-directional switching and 
does not require extensive modification to existing devices. Most desirably, the present 
disclosure finds application in personalized television. 

1 5 The present invention includes a method of preparing a plurality of data streams to allow 

seamless switching between the data streams by a switching device. The switching device may 
include data stream buffering for an output data stream. The method includes providing a 
plurality of data streams, where the data streams include data which is divided into segments that 
include synchronized starting points and end points on all of said plurality of data streams. The 

20 method includes providing gaps in the data streams between the end points and starting points, 
and increasing a data rate of the data streams at a time before an end point of a segment. This 
increase of the data rate can be performed by a number of techniques, alone or in combination, 
including multiplexing, variations in multiplexing, compression, changes in QOS (quality of 
service), using additional data streams, etc. The present invention includes switching from one 

25 of the plurality of data streams to another one of the data streams at an end point of a segment. 
Trigger gap indicators can be inserted in the data streams proximate the end points, to indicate to 
a switch that a switch point is present or imminent. 
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An illustrative embodiment of the present invention is used to encode MPEG-2 transport 
streams with multimedia data signals. This allows a receiver, such as a digital set top box, to 
seamlessly switch between multiple channels and produce output with no switching artifacts. 

U.S. Patent 5,913,031 issued to Blanchard describes an encoder for system level buffer 
5 management. However this patent describes maintaining a substantially constant data stream 
rate, with minor long-term adjustments to keep a post-switch frame buffer(s) full. This patent 
uses complex signal data rate analysis to maintain full frame buffers, and does not disclose a 
feature of adjusting a data rate to momentarily increase storage in the frame buffers. Further, this 
patent does not teach the creation of gaps in a data stream. Finally, this patent focuses on 
1 0 maintaining full buffers only through compression of data. 

An advantage of the present invention is a transmission system and method of seamlessly 
switching between a plurality of data streams to produce an output data stream without any 
switching artifacts. 

Another advantage of the present invention includes a system and method for seamless 
15 switching using presently deployed receiver systems such as digital set top boxes (STB). No 
extra buffering is required to be added to present receivers. 

Another advantage of the present invention includes a system and method for seamless 
switching that facilitates multi-directional switching and does not require extensive modification 
to the existing devices. 

20 Yet another advantage of the present invention includes a system and method for 

seamless switching that can be employed with personalized television applications. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, referred to herein and constituting a part hereof, illustrate 
the exemplary embodiments of the system and method for seamless switching of the present 
25 invention and, together with the description, serve to explain the principals of the invention. 

Fig. 1 is a block diagram of some components of a digital receiver; 

Fig. 2 shows switching in one direction between time synchronized data streams; 
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Fig. 3 shows the time synchronized data streams of Fig. 2 with an included switching 

delay; 

Fig. 4 shows the time synchronized data streams of Fig. 3 in accordance with the present 
invention; 

5 Fig. 5 is a flow diagram for transport stream generation according to an illustrative 

embodiment; 

Fig. 6 is a block diagram of a switch gap inserting multiplexer according to an illustrative 
embodiment; 

Fig. 7 is a block diagram of a multiplexed data stream; 

10 Fig. 8 shows the multiplexed data stream of Fig. 7 as modified by an illustrative 

embodiment; and 

Fig. 9 shows a transport stream with a personalized message according to an illustrative 
embodiment, 

DETAILED DESCRIPTION 

15 The present invention finds utility in various data transmission applications including 

analog and digital broadcasting, cable, cellular, satellite and terrestrial broadcasts, Video on 
Demand, Digital Versatile Disc (DVD), Internet, ethernet, wireless, ATM, MPEG (including 
MPEG 1, 2, 4 and variations thereof), AC3 etc. 

When switching from one source to another, the physical switch takes time, both in 
20 hardware, but also in software. In a digital television receiver for example, the input streams for 
audio and video are time multiplexed with other information into a transport stream. This time 
multiplexing makes it necessary to send the audio and video data in bursts and ahead of the 
presentation time. The data is buffered in the receiver 20 Fig. 1 and played out at a 
predetermined moment relative to a presentation clock. Because of the way that digital television 
25 receivers are constructed, the switching mechanism that allows the selection of video and audio 
streams from the transport stream is located before the playback buffering. And because the data 
is transmitted in burst-mode, there is no way to know the fill level of the buffer without detailed 
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knowledge of the incoming transport stream, and the current playback time of the receiver. There 
is also no way to know where the other audio and video stream transmissions are in relation to 
the current one. 

The way MPEG video is encoded makes it necessary to switch at the start of a video 
5 sequence, because otherwise the receiver has to wait for the start of the next sequence (for 
example, wait for the next I-Frame). A similar problem exists for audio, where if packets are 
missed, the audio sequence may be able to recover, but not without causing switching artifacts. 
All this makes the exact moment of switching over very critical, while there is no information 
embedded in the transport stream to find out what is being received in relation to the presentation 
10 time. The latency of the receiver processing system is also too big to react on what is being 
received without knowing ahead of time what will be coming. The conclusion can be that 
without the proper preparation of the transport stream to give the receiver time to react, the 
results of a switch will be non-deterministic. 

Fig. 2 shows one-directional switching between two data streams. For this example, 
15 consider two continuous MPEG-2 video streams, 50A and SOB, where a cross over point X 52 is 
defined at time T 54. Because the physical switch in the receiver takes time (d) 56, frame 1 in 
stream SOB is not received completely, and cannot be presented. The next completely received 
frame will be frame 2 of stream SOB (assuming that d is smaller than one frame transmission 
time, which is true for present switches). However, if frame 1 was the start of a sequence, then 
20 frame 2 may not be able to be presented because frame 2 depends on frame 1 . For example if 
frame 1 is the start of a sequence, then frame 1 might be an I-Frame (intrapicture frame). I- 
Frames are coded independently of all previous or future frames. Frame 2 would be a P-Frame 
(predictive frame), which is coded based on previous I-Frames or P-Frames; or B-Frame 
(bidirectionally predictive frame), which is coded based on the next and/or previous frames. If 
25 frame 1 was not received, then frame 2 could not be displayed, or any subsequent P-Frames or 
B-Frames until the next I-Frame is received. 

To get the cross over point to work, the transmission of frames in SOB is delayed in 
relation to 50A by at least the switching time (d) 56 to give the receiver time to react. This is 
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shown with added delay 57 Fig. 3 to stream SOB. There also is an indicator 58 Fig. 3 to signal to 
the receiver that the cross over point has been reached. 

While this solution will work for the switch from 50A to SOB, it will not work for a 
switch from SOB to 50A at X 52. The solution will not work if X 52 is a bi-directional cross over 
5 point, because a system would also have to delay 50 A with respect to SOB by at least switching 
time d 58 and that is not possible. The way to achieve this is to delay the transmission of the 
frames after the cross over point X 52 for both streams 50A and SOB without delaying the frames 
before the cross over point X 52, as illustrated in Fig. 4. 

The present invention can exploit burst-mode transmission and the time presentation of 
10 the digital television system to solve the problem. By preparing the transport stream in such a 
way that the playback buffer is filled with frames from before the cross over point X 52, the 
buffered frames can bridge the switching delay 56. So instead of delaying the transmission, an 
embodiment of the present invention sends frames ahead of time and uses the buffer and 
presentation timing to its advantage. 

1 5 Thus, by utilizing the independence between reception and presentation of the digital 

television systems, the present invention introduces a discontinuity in the transmission of all 
streams and creates a situation where a switch make a switch from 50A to SOB and from SOB to 
5 OA, a bi-directional cross over point. The number of parallel streams at the cross over point 
does not matter, thus the present invention can be scaled up to any number of streams and is in 

20 fact a multi-directional seamless switch point. 

Although described in terms of MPEG-2 encoded transport streams, the present invention 
will work with any type of data streams, in any situation wherein some control is provided over 
the data stream rates, such control being for example varying the bandwidth, varying the packet 
send rates, modifying the data stream multiplexing, compressing the data in the data streams, 
25 compressing the content streams etc. 

One embodiment of the present invention includes modifications to the layout of the 
transport stream, and transport stream generation equipment. This includes the addition of safe 
exit and safe entry points at the right points in the transport stream. The conditions for these 
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points exactly match the conditions for standardized splicing points, so standard encoding 
equipment that supports splicing points can be used to generate the right streams. The addition of 
the transmission delay gap is not standard, although its creation is not unlike the operation of a 
statistical multiplexer which, based on certain input parameters, can vary the bandwidth assigned 
5 to individual streams. For this embodiment, the bandwidth should be increased in the period 
before the gap, in order to pre-send all of the information that would have been sent during the 
gap if the stream distribution was constant. When that information has been sent, the bandwidth 
must be reduced to zero to stop the transmission of that stream completely, thus creating the 
switch gap. 

10 An illustrative embodiment will now be presented for how an MPEG-2 transmission 

system or relay system (like a cable head-end) can be extended to provide the outgoing transport 
stream that supports seamless switching. While the diversity among MPEG-2 receivers is 
relatively small for the MPEG-2 processing part, the diversity among transmission systems can 
be very big. For that reason this description focuses on the basic elements involved in the 

1 5 creation of a seamless switch compatible transport stream, to allow one skilled in the art for a 
specific transmitter arrangement to adapt the techniques to that specific transmitter arrangement. 

Fig. 5 shows a flow diagram with the steps involved in the creation of a seamless switch 
ready transport stream. Multiple content material input streams 71 are prepared, for example in 
parallel. Note that an actual implementation may combine steps or split them up differently 
20 based on the architecture of the actual equipment involved. As long as the resulting transport 
stream has the correct elements, any implementation is valid. 

The first step is content preparation 72. In order to allow seamless switching at the 
switch point the content material is prepared with correct safe entry and safe exit points. The 
encoding at these points is equal to splicing points, but the splicing point count down and 
25 indication does not necessarily have to be present for seamless switching, since other triggers 
will indicate the switching moment. Because, after the switching moment, all parallel streams 
will have to start with an I-Frame, a burst of data across the transport stream is expected at that 
point. It is possible to limit the quality of these first I-Frames, or use fixed rate multiplexing. 
Because of the interaction between the individual streams with respect to the overall transport 
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stream bandwidth, it may be easier to combine content preparation with switch gap creation and 
multiplexing. This depends on the overall architecture of the equipment. 

Content preparation 72 can be an offline process, taking place well in advance of actual 
transmission. It can also be mixed, with one or multiple streams being processed in real-time, 
5 while other streams are prepared offline. 

The next step is switch gap creation and multiplexing 74. In order to allow frame 
accurate seamless switching in all directions between several different streams, special 
multiplexing requirements are provided. First of all the switch gap is created. The requirement 
for the stream around the gap is that all the information that is necessary for the duration of the 

10 gap is transmitted before the gap. This is achieved by having the data available ahead of time. 
One method is if the multiplexer has on-demand control over the influx of data, and can request 
extra data at any time. To create this gap in a continuous stream of data without control over the 
source of the data, a delay buffer 75 is used. This buffer 75 continuously delays the stream by a 
fixed amount of time and thus creates a limited amount of read ahead control over the incoming 

1 5 data. On demand the multiplexer can read the data in the delay buffer faster than the continuous 
data rate, as long as the faster read is compensated afterwards. Since the average data rate over 
the speed-up and the gap is the same as the continuous data rate, there is no problem. The end 
result is a continuously variable delay line with a control unit that follows a preset delay pattern 
when triggered. 

20 The time needed for the gap is the time needed for the receiver to switch, and depends on 

the hardware and software of the receiver. In the case of STB receivers, the gap timing may vary 
based on the brand of STB. Interestingly enough, since a particular television distribution 
network (such as a cable headend system or neighborhood distribution network) typically 
depends on all STBs being the same type, the present invention allows gap timing to be adjusted 

25 at a television network distribution network to match the requirements of the common STB in 

use on that distribution network. Other distribution networks can have gap timing commensurate 
with the STBs on those networks. Alternatively, the gap timing may be adjusted to 
accommodate the slowest switching time of a set of different STBs. 
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Fig. 6 outlines an example gap creating multiplexer 60. This multiplexer 60 combines 
features of both steps 74 and 76 of Fig. 5, since trigger insertion 76 is closely timed to the switch 
gap creation 74. The gap trigger 77 Fig. 6 causes the gap delay components 78 to delay the 
content material input streams 71 for the duration of the gap time. Also, trigger insertion 62 
5 occurs to insert a trigger message 58 into the input streams at the switching point. The trigger 
message 58 can have different formats. It can be a separate private PES packet time 
synchronized with the content streams. The trigger message 58 can also be an indicator packed 
into one of the content streams. Basically, anything that can be detected by the receiver and has a 
fixed timing relation with the Switch Gap can be used as a Trigger Message 58. 

10 Content selection info insertion is an optional part of this step 74 Fig. 5. Depending on 

the application that makes use of the seamless switching facilities, special content selection 
information may need to be inserted into the outgoing transport stream. This information has a 
time relation with the switch gaps, but only in the sense that the selection information must be 
available before the choice has to be made and must have a mapping to the corresponding switch 

15 gap. Typically the actual information in this data stream is completely dependent on the 
application. 

Fig. 7 shows a sample multiplexed data stream 120. This multiplexed data stream for 
example can be a transport stream according to MPEG-2 standards. The multiplexed data stream 
120 includes packets that are time multiplexed together, as shown by packets A-E, indicated by 
20 label 122. Each of the packets A-E are for a different content stream. The data flow is 

continuous, as shown by arrow 125. The bandwidth of the multiplexed data stream 120 is higher 
than required to supply the packets 122, therefore empty transport stream packets 124 are 
inserted. These packets 124 may be "empty", or carry other data separate from the content 
stream packets 122. 

25 Fig. 8 shows the multiplexed data stream 120 at a point just before and after a gap 57 is 

inserted, in accordance with an illustrative embodiment. The packets 122 are still sent, but to 
increase the data rate, the empty transport stream packets 124 are reduced or eliminated at a time 
before the gap 57. This increases the data delivery rate for the content streams provided by the 
packets 122. Just before the gap 57, a switch trigger message 58 is inserted. As previously 
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described, the switch trigger 58 may be a special (possibly empty) packet which is inserted by 
the switch gap inserting multiplexer 60 Fig. 6 for example, or the switch trigger 58 may be an 
indication on another packet. In this embodiment, the gap 57 comprises empty (NIL) packets. 
One way for a time division multiplex (like an MPEG-2 transport stream) to avoid sending 
5 relevant data is to send something else, so to create a gap in the data streams A-E, different 
packets are sent, either packets from unrelated streams, or empty packets. After the gap 57, the 
data delivery rate is decreased by again inserting the empty transport stream packets 124. 

The following calculation example provides details assuming an MPEG-2 transport 
stream (188 bytes per packet) and the requirement of a 30 ms switch gap (see below). It also 

10 assumes 4 Mbps for the content streams. To create a gap of about 30 ms in a 27 Mbps transport 
stream, 540 transport stream packets must be cleared ((27,000,000 / (8 * 188)) * 0.030). If there 
are five parallel 4 Mbps (roughly 81 TS packets in 30 ms) streams that need to have the switch 
gap at the same time, then 5*81=405 packets (N) of those five streams together must be moved 
forward (earlier) out of the switch gap area. All other content in the transport stream can be left 

15 in the gap; only the relevant streams need to be moved out. Alternatively, other information can 
be placed in the empty packets in the gap. The available spare bandwidth in the transport stream 
determines the time over which the streams transmission would have to be condensed. With a 
spare bandwidth (S) specified in bps, the amount of time needed to create the 30 msec gap would 
be(N* 1504)/ S. In this case it would be 405* 1504/4000000= 150 msec. 

20 Therefore with this transport stream configuration the illustrative embodiment would 

need to start sending information ahead about 180 msec before the start of the next fragment. 
That way it would finish sending all the information up to the start of the next fragment 30 msec 
before the actual start of that fragment, thus creating a 30 msec gap. Note that the maximum 
amount of transport stream packets that needs to be moved is bound by the overall transport 

25 stream bandwidth. 

The actual required gap depends on the response speed of the software in the receiver 20. 
A gap of 30 msec (roughly one frame) is assumed to be the typical maximum needed for most 
receivers. 
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An illustrative embodiment for the transport stream generation device and multi-direction 
seamless switching is the personalization of TV commercials in a digital television environment. 
In this application a personalized commercial would consist of several sequential fragments 
(slots), each having several parallel options. All options for a slot would be transmitted 
5 simultaneously within the same transport stream and, at the beginning of each slot, a decision 
would be made by the receiver which of the options for this slot to show to the viewer. The 
personalized ad would be inserted into the flow of the main program in a time slot that would be 
big enough for the total personalized ad (typically 30 seconds). Within the main program a 
trigger would be present to tell the ad insertion equipment to insert the personalized ad. The ad 
1 0 insertion equipment would be constructed around a transport stream generation device as 

described in the previous section, an ad store to keep the content fragments for all the options of 
all the slots of the ad, and a management process that controls the overall insertion process when 
an insertion trigger is detected in the main program. To know which ad to insert at which point in 
time an insertion schedule is available. 

15 Fig. 9 shows a transport stream 90 prepared according to an embodiment of the present 

invention. In this case the content selection information added by the personalization application 
consists of the indicated SIM (Sequence Identification Message) 100, SOM (Sequence Option 
Message) 102 and SEM (Sequence End Message) 104, while the switch point trigger message is 
indicated by the SPM message 106. In this example the personalized ad comprises two segments 

20 101, which have multiple choices of media data, and are preceded by gaps 57 to allow for 

switching time to an appropriate media data segment. The transport stream 90 shown indicates a 
personalized ad inserted into a main program while that main program is the only one present in 
the transport stream 90. By means of commercial break synchronization the same personalized 
ad can be used for all programs transmitted in the same transport stream. 

25 It will understood that various modifications may be made to the embodiments disclosed 

herein. Therefore, the above description should not be construed as limiting, but merely as 
exemplification of the various embodiments. Those skilled in the art will envision other 
modifications within the scope and spirit of the claims appended hereto. 

What is claimed is: 
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